System and methodologies for occupant monitoring utilizing digital neuromorphic (nm) data and fovea tracking

ABSTRACT

A system and methodologies for neuromorphic vision simulate conventional analog NM system functionality and generate digital NM image data that facilitate improved object detection, classification, and tracking so as to detect and predict movement of a vehicle occupant.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/726,883, filed 6 Oct. 2017, which is a continuation-in-part ofco-pending U.S. application Ser. No. 15/386,220, filed 21 Dec. 2016, nowU.S. Pat. No. 10,133,944, issued 20 Nov. 2018, the entire contents ofwhich being hereby incorporated herein by reference.

COPYRIGHT

A portion of the disclosure of this patent document contains materialwhich is subject to (copyright or mask work) protection. The (copyrightor mask work) owner has no objection to the facsimile reproduction byanyone of the patent document or the patent disclosure, as it appears inthe Patent and Trademark Office patent file or records, but otherwisereserves all (copyright or mask work) rights whatsoever.

BACKGROUND

The present disclosure relates to systems, components, and methodologiesfor image processing. In particular, the present disclosure relates tosystems, components, and methodologies that perform image processing forusing digital NeuroMorphic (NM) vision techniques.

SUMMARY

According to the present disclosure, systems, components, andmethodologies are provided for NM-based image data generation, imagedata processing and subsequent use to detect and/or identify objects andobject movement in such image data for assistance, automation, controland/or documentation.

In accordance with disclosed embodiments, structure and software areprovided for simulation of conventional analog NM system functionalityusing a digital NM vision system that incorporates at least one detectorthat includes one or more NM sensors, a digital retina implementedusing, for example, CMOS technology that enables generation of digitalNM data for image data processing by a digital NM engine thatfacilitates improved object detection, classification, and tracking. Assuch, exemplary embodiments are directed to structure and software thatmay simulate analog NM system functionality.

In accordance with at least one embodiment, the digital NM engine mayinclude a combination of one or more detectors and one or moreprocessors running software on back-end to generate digital NM output.

In accordance with at least one embodiment, the digital NM visionsystem, its components and utilized methodologies may be used tocompress high framerate video data by performing feature extractionclose to an imaging sensor to generate an encoded version of image datathat includes differences and surrounding spatio-temporal regions forsubsequent image processing. Thus, in accordance with at least oneembodiment, the hardware and methodologies may be utilized as aneffective method for compressing high framerate video, e.g., byanalyzing image data to compress the data by capturing differencesbetween a current frame and a one or more previous frames and applying atransformation.

In accordance with at least one embodiment, the digital NM vision systemand/or at least a subset of its components may be incorporated in astereo neuromorphic pair. In accordance with at least oneimplementation, components of the digital NM vision system may beincorporated in a compound camera. In such an implementation, thecomputational element of each imaging sensor may be coupled to othercomputational elements of other imaging sensors, e.g., adjacent sensorsor other types of sensors, to collaborate with other computationalelements to provide functionality. For example, in accordance with atleast one implementation, the digital NM vision system components may beincorporated in an event-based camera.

In accordance with at least some embodiments, post-processing operationsfor data generated by a digital NM detector are performed that generatedigital NM data output that enables image data processing for improvedobject detection, classification, and tracking.

In accordance with at least some embodiments, post-processing operationsinclude velocity vector and velocity trajectory generation as well asimage segmentation based on high density velocity vectors, generatingand analyzing spatial temporal patterns N×5D (x,y,Vx,Vy,Vz) and N×5D(x,y,Vx,Vy,t), generating composite spikes with neighborhood data,generating compound fovea generation, performing velocity segmentationusing compound foveas, computing velocity trajectories for fovea pixels,respiking foveas, computing velocity vectors by rotating velocity spaceand using previous frames to increase the temporal and spatialresolution of spikes, using spread functions to optimize search forvelocity vectors, performing a double sparsity approach to computevelocity vectors, and computing velocity trajectories using affinetransformations.

Disclosed embodiments pertain to components, system and methodologiesfor using digital NM data and components and systems for vehicleoccupant tracking. Disclosed embodiments may be implemented using thedigital NM engine in combination with micro fovea and composite spikesto perform fovea tracking.

Disclosed embodiments may use intermediate generated digital NM dataand/or generate relative velocity vector data to perform fine patternrecognition capabilities measured at close range including headtracking, eye tracking, biometrics, face detection, pupil dilation, eyegaze, color, blink and lid detection, coordinated head and eye tracking,unsupervised presponding, driver texting detection and alert, driverfatigue detection, occupant body gestures including hand gestures, seatbelt gesture detection, driver emotion detection, lip reading,handwriting recognition.

In accordance with disclosed embodiments, eye tracking may be performedby tracking eye gaze using generation and analysis of three dimensional(3D) foveas. Likewise, 3D foveas may be used to detect and track eye-lidmovement, determine eye color. In accordance with disclosed embodiments,eye blink and wink detection may be performed by generating andanalyzing spatial temporal patterns. Such embodiments may also enablecomponents of the system to be used to perform a methodology fordetecting driver fatigue or driver emotion and generating an alertregarding such a detection.

In accordance with disclosed embodiments, coordinated head and eyetracking can be effectively performed.

Additional features of the present disclosure will become apparent tothose skilled in the art upon consideration of illustrative embodimentsexemplifying the best mode of carrying out the disclosure as presentlyperceived.

BRIEF DESCRIPTION OF THE FIGURES

The detailed description particularly refers to the accompanying figuresin which:

FIG. 1 is an illustrative diagram of hardware structure and softwareutilized by disclosed embodiments to provide simulation of conventional,human eye, analog NM system functionality.

FIG. 2 illustrates an example of a methodology provided in accordancewith at least one embodiment that performs simulation of conventional,human eye, analog NM system functionality.

FIG. 3 is an illustrative embodiment for explaining limitations ofconventional video camera and is provided to explain its operation andlimitations.

FIG. 4 is an illustration of an exemplary operation of a gray-levelvideo camera.

FIG. 5 illustrates an exemplary configuration of a conventional colorvideo camera.

FIG. 6 illustrates an exemplary configuration and operation of analog NMcamera operation.

FIG. 7 provides a diagrammatic view of an output of a conventional videocamera (non-NM) that is stationary relative to the environment sensingan object that is also stationary relative to the camera.

FIG. 8 provides a diagrammatic view of the output of a conventionalvideo camera that is stationary relative to the environment and sensingtwo objects in the environment in a series of frames 0-4 moving from thelower-left to the upper-right at a high frame rate.

FIG. 9 provides an illustrative diagram for explanation of conventionalanalog NM camera operation.

FIG. 10 provides a diagrammatic view of the operation of a single analogneuromorphic pixel.

FIG. 11 demonstrates the advantages of a digital NM vision systemprovided in accordance with the disclosed embodiments over both aconventional, analog NM camera and a typical video camera.

FIG. 12 shows a conceptual, illustrative view of a digital NM sensorprovided in accordance with the disclosed embodiments.

FIG. 13 illustrates one example of conceptual components of the digitalretina provided in accordance with at least one disclosed embodiment infurther detail.

FIG. 14 illustrates conceptual, illustrative view of the functionalityprovided by components illustrated in FIG. 13.

FIG. 15 illustrates a transformation of the input image using acenter-on adaptive threshold.

FIG. 16 provides an explanatory illustration of the utility provided ingenerating a micro-fovea spike sequence.

FIG. 17 provides an explanatory illustration of the utility provided inidentifying spatio-temporal events via motion patterns within generatedsequences of NM images.

FIG. 18 illustrates an illustrative diagram useful in describing oneexemplary process of constructing a velocity space.

FIG. 19 illustrates one example of how spike data may be associatedtogether to determine their velocity.

FIG. 20 illustrates one example of how disclosed embodiments may beutilized to track and/or search for movement of a vehicle driver's headand/or eye(s).

FIG. 21 illustrates one example of how disclosed embodiments may beutilized to identify and/or predict an action or event associated withmovement of an occupant of a vehicle driver.

FIG. 22 illustrates one example of how disclosed embodiments may beutilized to detect driver texting on a mobile phone and generate analert regarding such and similar operations.

DETAILED DESCRIPTION

The figures and descriptions provided herein may have been simplified toillustrate aspects that are relevant for a clear understanding of theherein described devices, systems, and methods, while eliminating, forthe purpose of clarity, other aspects that may be found in typicaldevices, systems, and methods. Those of ordinary skill may recognizethat other elements and/or operations may be desirable and/or necessaryto implement the devices, systems, and methods described herein. Becausesuch elements and operations are well known in the art, and because theydo not facilitate a better understanding of the present disclosure, adiscussion of such elements and operations may not be provided herein.However, the present disclosure is deemed to inherently include all suchelements, variations, and modifications to the described aspects thatwould be known to those of ordinary skill in the art.

Exemplary embodiments are directed to structure and software forsimulation of conventional NM system vision functionality, e.g., thatprovided by the human eye.

Commercially available image detection and processing equipmentroutinely use solid-state detectors to capture large numbers of frameseach second. By displaying those images at high speed, the viewer hasthe illusion of motion. This is the basis of recorded video images.

However, when such video data is analyzed by computers running imageprocessing and analysis software, the large number of frames used togive the impression of motion can overwhelm the computational capabilityof the computers. This is because a high frame rate video may provide somuch data that the computer is incapable of analyzing the data becausethe data is changing too quickly. Conventionally, efforts have been madeto increase the ability for image processing by increasing theprocessing speed of processors analyzing the image data.

Additionally, analog-based Neuromorphic (NM) processing techniques havebeen developed that mimic or simulate the human eye. NM processingrelies on the idea that it is not necessary to analyze all of the dataincluded in a video image; rather NM prioritizes analysis on determiningthe changes that occur in the image data while de-prioritizing the imagedata that remains the same from frame to frame because the non-changingdata is redundant.

More specifically, by mimicking operation of the human eye and brain,processors and software can capture and identify image data of interest,spatial and temporal changes, and output that data for labor intensiveimage processing that enables all aspects of image processing,automation and assistive control, analysis and diagnostic systemsutilizing image processing.

This requires the ability to continuously track and record pixelamplitudes for only those pixels amplitudes changes above a prescribedthreshold. Conventionally, this approach has been implemented usinganalog NM cameras; however, application of such technology provides higheffective frame rates but with spatial image sizes and spatialresolutions due to the extra cost of analog processing embedded intoeach pixel of the imager. Thus, there is no conventional mechanism toeffectively use NM image processing for real-time acquired image datahas yet to be successfully implemented.

To the contrary, the presently disclosed, digital NM vision system, itscomponents and utilized methodologies have been developed to performingfeature extraction from frame data in a way that enables an imagingsensor(s) to generate an encoded version of image data that includesonly data indicating differences indicative of movement and surroundingspatio-temporal regions for subsequent image processing. In turn, thisfurther improves the accuracy and throughput of the system.

Illustrative embodiments perform post-processing operations for datagenerated by a digital NM detector to generate digital NM data outputthat enables image data processing for improved object detection,classification, and tracking.

An example of such a digital NM detector is illustrated in FIG. 1,including structure and software are provided for simulation ofconventional analog NM system functionality using a digital NM detector110 that incorporates a digital retina implemented, for example, usingCMOS technology. Thus, as shown in FIG. 1, a digital NM vision system100 may include a digital NM detector 110 that receives rays of light105 reflected from objects in an environment. Those rays of light may bedetected by a sensor, e.g., photoreceptor or array thereof, 120 that maybe configured to convert those rays of light in image data 125,including images of the objects in the environment. For example, eachphotoreceptor may convert the light energy into, e.g., proportionalanalog intensity values.

As explained in more detail herein, that image data may be output fromthe sensor array 120 into a digital retina 130 that converts that imagedata into “spikes” using various image processing and data processingtechniques. However, the digital retina 130 includes digital circuitrythat generates spike data indicative of a spike in association with aparticular photoreceptor within the sensor array 120 whenever theintensity value measured by that photo receptor exceeds a threshold. Thedigital retina 130 may be implemented using various solid-statetechnology including, for example, ComplementaryMetal-Oxide-Semiconductor (CMOS) implemented technology, e.g., one ormore Field Programmable, Gate Arrays (FPGAs), (GPUs) or functionally orstructurally similar devices integrated circuits and associated softwareand/or firmware provided in, for example, Application SpecificIntegrated Circuits (ASICs). Spike data is generated not only based onthe data generated by that particular photoreceptor but also can takeinto account data generated by neighboring, nearby or near photoreceptors, e.g., one photoreceptor away so as to simulate operation ofspecific parts of the human eye that utilize communication betweenphotoreceptors when formulating spike data. Spike data 135 generated bythe digital retina 130 is input into one or more velocity transformationmodules 140 that generate velocity vectors 115 for subsequent analysisby the digital NM engine 145.

Additionally, the digital retina 130 generates, outputs and exchangesvarious data that enables digital NM vision including spike (sparse)data, 5D (x, y, t, Vx, Vy) velocity data and other digital data that isgenerated by or analyzed by the digital NM engine 145. Each spikespecifies its spatial location within the input image (x, y), itstemporal coordinate or timestamp (t), and its optical velocity (Vx, Vy).

Thus, in accordance with at least some embodiments, post-processingoperations include velocity vector and velocity trajectory generation aswell as image segmentation based on high density velocity vectors,generating and analyzing spatial temporal patterns N×5D (x,y,Vx,Vy,Vz)and N×5D (x,y,Vx,Vy,t), generating composite spikes with neighborhooddata, generating compound fovea generation, performing velocitysegmentation using compound foveas, computing velocity trajectories forfovea pixels, respiking foveas, computing velocity vectors by rotatingvelocity space and using previous frames to increase the temporal andspatial resolution of spikes, using spread functions to optimize searchfor velocity vectors, performing a double sparsity approach to computevelocity vectors, and computing velocity trajectories using affinetransformations.

In this way, the disclosed digital NM vision system including a digitalNM detector that includes a sensor array (including individual sensors),digital retina and velocity transformation modules 140. The digital NMvision system also can include a digital NM engine 145 that performimage and data processing operations on the velocity vector datagenerated by the digital NM detector that enables image data processingfor improved object detection, classification, and tracking, includingmachine and deep learning. As such, in accordance with at least oneembodiment, the digital NM engine 145 may include one or processorsrunning software to generate digital NM output data for analysis andsubsequent control of components with the environment imaged by thedetector 110. Operation of the digital NM engine 145 is furtherdiscussed herein with connection to FIGS. 17-20.

Appendix A includes an example of one implementation of software codeutilized for generation of spike data and associated to velocity vectordata.

As used herein, the term “velocity vector” refers to a mathematicalrepresentation of optical flow of pixels (or photoreceptors) in imagedata. Velocity vector data may be used to characterize or represent avelocity space, which may be thought of as the spatial and temporalrepresentation of video data including a plurality of frames depictingmovement of an object in an environment.

In velocity space, pixels having the same velocity vector may beaggregated and associated with one another to perform velocitysegmentation. As explained herein, velocity segmentation enables theability to identify and differentiate objects within the image databased on their relative motion over frames of image data. Thus,disclosed embodiments pertain to components, system and methodologiesfor generating relative velocity vectors using digital NM data andcomponents and systems for utilizing those velocity vectors for imageprocessing, object detection, classification, and tracking.

More specifically, disclosed embodiments pertain to components, systemand methodologies for using digital NM data and components and systemsfor vehicle occupant tracking.

In accordance with at least one additional embodiment, digital NMdetector output may include velocity vector data that indicates oridentifies basic features (e.g., edges) of objects included in the imagedata.

Unlike conventional imaging processing technology, this identificationof basic features may be performed at the fovea stage. A fovea (or morespecifically, the foveal centralis) in anatomical terms, is the smalldepression in the retina of the eye where visual acuity is highest. Thecenter of the field of vision is focused in this region, where retinalcones are particularly concentrated. As a result, the fovea providesacute central vision that enable humans to perform activities thatrequire significant visual acuity. Within the human eye, the fovea issurrounded by decreasingly dense layers of sensory cells so as toprovide ever decreasing resolution data on the periphery of the humanfield of vision.

Disclosed embodiments may be implemented using the digital NM engine incombination with micro fovea and composite spikes to perform foveatracking.

In machine implemented image processing, the term “fovea” has been usedto refer to a shape that corresponds to detection of a shape of anobject of interest that enables tracking of the eye's foveal centralison that object. By generating foveas for objects, the digital NM visionssystem is performing operations akin to “smooth pursuit” eye movement inthe human eye. Smooth pursuit eye movements allow the eyes to closelyfollow a moving object. It is one of two ways that visual animals canvoluntarily shift gaze, the other being saccadic eye movements. As aresult, the term “fovea” is used herein to refer to the edge datagenerated by the digital NM vision system because that data is used bythe human eye to center the human eye's fovial centralis on an objectfor object tracking.

It is well understood that the human eye can only perform smooth pursuiteye movement with regard to one object. To the contrary, in accordancewith the presently disclosed embodiments, smooth pursuit sensor movementmay be used to track a plurality of objects simultaneously.

With this understanding of fovea data generated by the digital NM visionsystem in mind, it should be appreciated that further data details maybe necessary to document what an object looks like. Those furtherdetails may require recording and analyzing pixel data surrounding eachspike within the fovea. This may enable the ability to identify color,texture, etc. This neighborhood of pixel data surrounding each spike maybe conceptually thought of as a “micro-fovea” because generation of anobject's fovea may be performed by aggregating the micro-fovea data. Inthis way, foveas are created by temporal and spatial aggregation ofmicro-foveas along a velocity profile.

As a result, micro-fovea can be linked together to define one or moreboundaries between foreground objects and background, thus creatingvelocity silhouettes. Each velocity silhouette defines an edge at theboundary between a foreground object and a background object. In thisway, intra-object fovea trajectories (i.e., for more than one object)indicate shape change for non-rigid objects.

Thus, micro-foveas include the spike data generated by an object'smotion and pixel data surrounding each of the spikes within that data.Thus, each object within an image sequence might have a fovea generatedfor it which indicates the motion of each object during that imagesequence, as indicated by a plurality of microfovea.

An example and description of how to formulate fovea trajectories isprovided in Appendix A.

Depending on what the object is, image processing may be altereddepending on the type of object being tracked. The further detailsprovided by the micro-fovea may be analyzed to enable the ability toidentify color, texture, etc.

Disclosed embodiments may use intermediate generated digital NM dataand/or generate relative velocity vector data to perform fine patternrecognition capabilities measured at close range including headtracking, eye tracking, biometrics, face detection, pupil dilation, eyegaze, color, blink and lid detection, coordinated head and eye tracking,unsupervised presponding, driver texting detection and alert, driverfatigue detection, occupant body gestures including hand gestures, seatbelt gesture detection, driver emotion detection, lip reading,handwriting recognition, as explained herein.

Likewise, in accordance with disclosed embodiments, eye tracking may beperformed by tracking eye gaze using generation and analysis of threedimensional (3D) foveas. Thus, as explained herein, 3D foveas may beused to detect and track eye-lid movement, determine eye color.

In accordance with disclosed embodiments, eye blink and wink detectionmay be performed by generating and analyzing spatial temporal patterns.Such embodiments may also enable components of the system to be used toperform a methodology for detecting driver fatigue or driver emotion andgenerating an alert regarding such a detection. Further, in accordancewith disclosed embodiments, coordinated head and eye tracking can beeffectively performed.

Post-processing operations for data generated by a digital NM detectormay be performed to generate digital NM data output that enables imagedata processing for improved object detection, classification, andtracking. More specifically, once edges of an object are detected usingspike data, additional analysis may be performed on the image data atthe pixel level to improve the quality of the image. In other words, themotion present in an image or sequence of images (as identified byspikes) is used to identify objects of interest that may be subqeuentlyanalyed at the pixel level to provide additional information.

Thus, although FIG. 1 illustrates the digital NM engine 145 as receivingoutput from the digital NM detector 110, it should be appreciated thatthe digital NM engine 145 may provide data to the digital NM detector110 as well as receive input from the detector 110. Such connections andcommunication may be utilized to perform machine learning for thedigital NM detector 110 to facilitate further functionality and/orinteroperability and/or improve upon functionality, analysis andoperation of digital NM detector 110.

As explained above, the motion present for an object during an imagesequence may be represented using spike data, which may be analyzed todetermine velocity vector data. That velocity vector data may then beanalyzed by the digital NM engine 145 to identify edges of the object.Velocity segmentation of the image data using the vector velocitiesgenerated from the spike data may be used to generate edge data. Byaggregating the edge data of the object, a representation of the object,i.e., a fovea, may be produced. Conceptually, the fovea is made up ofthe trajectory of velocity segmentations for the object over time.

FIG. 2 illustrates an example of a methodology that performs simulationof conventional, human eye, analog NM system functionality. As shown inFIG. 2, the methodology begins at 200 and control proceeds to 205 atwhich rays of light reflected from objects in an environment aredetected by a sensor. Control then proceeds to 210 at which those raysof light are converted into image data including images of the objectsin the environment, e.g., represented by proportional analog intensityvalues.

Control 215, at which that image data is output into a digital retina.Subsequently, at 220, the image data is converted into “spikes” based onthe data generated by a particular photoreceptor but also taking intoaccount data generated by neighboring, nearby or near photoreceptors,e.g., one photoreceptor away so as to simulate operation of specificparts of the human eye that utilize communication between photoreceptors when formulating spike data. Control then proceeds to 225, atwhich spike data generated by the digital retina is input into one ormore velocity transformation modules to generate velocity vectors.Control then proceeds to 230, at which the velocity vector data isanalyzed to determine spatio-temporal patterns resulting fromspatio-temporal events to perform object detection, classification, andtracking, including machine and deep learning, e.g., includingidentifying edges of objects by velocity sementation of the image datausing the vector velocities generated from the spike data. Byaggregating the edge data of the object, a representation of the object,i.e., a fovea, may be produced. Conceptually, the fovea is made up ofthe trajectory of velocity segmentations for the object over time.

As explained in connection with FIGS. 10-16, one key difference betweenconventional, analog NM systems and the structure and methodologiesutilized by the disclosed embodiments is that conventional, analog NMcameras generate a single spike for each pixel or photoreceptor includedin the camera. Thus, each pixel provides data for generation of acorresponding spike in a one-to-one relationship. As a result, analog NMsystems require high frame rates to generate sufficient spike data forsubsequent analysis of edges and the like.

To the contrary, in accordance with the presently disclosed embodiments,the output data generated from individual pixels or photoreceptors arecombined and considered when generating spike data. Thus, data generatedby pixels located near a particular pixel may be taken intoconsideration when generating spike data for that particular pixel. As aresult, and explained below in more detail, the generated spike datatakes into consideration data generated by a neighborhood of pixels. Thetechnical effect of this distinction is far reaching for accuracy,processing speed and image data applications.

However, prior to documenting the technical utility and effect of thepresently disclosed digital NM vision system over conventional analog NMcameras, an explanation of the functionality and limitations ofconventional image processing techniques provides technical context.Accordingly, a brief explanation of conventional image processingtechniques and the corresponding limitations and utility of thosetechniques is provided so as to highlight the technical complexity of NMvision and the utility of the presently disclosed implementation ofdigital NM vision.

Conventional image processing is based on frame-by-frame processing andexcludes temporal information in the scene. For example, conventionalimage processing of video data processes each frame of a video sequencewhile excluding temporal information. Thus, while an input photographicimage may provide data relevant to analyzing a scene depicted in theinput photographic image, conventional image processing systems do notor cannot utilize all the data provided in the image. This severelylimits the ability for image processing systems to analyze temporal dataincluded in image data.

For example, conventional pixel-level labelling tasks, e.g., by semanticsegmentation used conventionally for image recognition, have a limitedcapacity to differentiate and delineate objects within an input image.Accordingly, the limitations of such approaches also impede the abilityto use deep learning techniques for image recognition and objecttracking. The technological limitations resulting from that omission andthe value of taking into consideration the relationship between spatialand temporal information and resolution is discussed further herein inconnection to FIG. 11 herein.

Conventionally, segmentation using Convolutional Neural Networks (CNNs)and Conditional Random Fields (CRFs)-based probabilistic graphicalmodelling has been used to formulate mean-field approximate inferencesfor CRFs with Gaussian pairwise potentials as Recurrent Neural Networks(RNN). Such networks, conventionally referred to as CRF-RNNs, have beenused to improve image processing using deep networks. However, theresulting image data is still lacking. Likewise, ground truthsegmentation data for the same input image data may also be lacking.

FIG. 3 further illustrates the limitations of conventional video camera320 and is provided to explain its operation and limitations. As shownin FIG. 3, a video camera 320 senses rays of light 305, 310 reflectedfrom an object 300 in an environment. A lens 315 focuses the rays oflight onto a photoreceptor array 325 and the camera 320 may includefilters for filtering the light as well. The photoreceptor array 325converts the light energy into a spatial digital image. As a result ofcamera operation, a temporal sequence of images 330 representing changesin the environment is output. This is the conventional operation ofvideo cameras in general which results in a temporal sequence of spatialimages with no mechanism to differentiate subsequent analysis of imagedata included in that sequence of frames.

As illustrated in FIG. 4, a gray-level video camera focuses rays oflights 400 onto a surface of photoreceptors 410. The surface 410 of thephotoreceptor array sensor 415 is subdivided into bins (or pixels). Theamount of light accumulated at each bin (or pixel) of the photoreceptor410 is converted to a digital value 420 and stored in digital image 425at the pixel corresponding to each photoreceptor bin (or pixel).

Likewise, FIG. 5 illustrates an exemplary configuration of aconventional color video camera. In operation, like in FIGS. 3 and 4,rays of light 500 reflected from an object in an environment areincident on a surface of a photoreceptor array 505. However, filtersincluded in the color video camera include filters which divide theincident light into color components (e.g., red, green, and bluecomponents). The filtered rays of light are then focused on to thephotoreceptor array, in the same way as FIG. 4, and the photoreceptorarray converts the color components into a digitized color image 510 foroutput. In this way, the sensor array 505 converts color components intoa color image 515.

To better understand the innovative concepts disclosed herein withrelation to NM vision, a brief explanation of analog NM camera operationis now provided. As shown in FIG. 6, use of analog 2D NM camerasproduces output data 605 from relative movement that is sparse buthighly relevant. As a result, the static, non-changing parts of an inputimage 600 may be automatically excluded and only relevant imageinformation related to changes in the scene may remain as shown at 605and 610. Because, analog 2D NM cameras extract and output relevant andsparse motion information, such NM cameras have particular utility forearly detection of start/change of movement. Further, analog 2D NMcameras are effective for use in logarithmic pre-amplification of imageintensity, temporal-gain adaptation, edge detection, and 2-D motiondetection at the imaging plane.

In contrast, FIG. 7 provides a diagrammatic view of an output of aconventional video camera (non-NM) that is stationary relative to theenvironment sensing an object that is also stationary relative to thecamera. As shown in FIG. 7, the camera outputs a sequence of imageframes, depicted as gray squares, 705, with the same size (width andheight) at a high frame rate in frames 0-4. Because the object and thecamera are stationary, the video camera outputs similar and redundantimage frames at a high frame rate resulting in a high data rate but noadditional spatial data content other than what was included in frame 0.

This issue of a large amount of data is further compounded whenattempting to use a fast frame rate to compute motion vectors forcomputer vision applications. As shown in FIG. 8, a diagrammatic view isprovided of the output of a conventional video camera that is stationaryrelative to the environment and sensing two objects in the environmentin a series of frames 0-4 moving from the lower-left to the upper-rightat a high frame rate. The conventional video camera outputs a sequenceof images depicting the movement of each object.

Assuming the image sequence is arbitrarily subdivided into key frames800 (frame 0), 800′ (frame 4) and tween frames 810 (frames 1-3), then,intuitively, detecting and tracking the objects' motion appearsstraightforward by viewing the key frames along with the tween frames.This is particularly true when the spatial resolution (width and height)and the temporal resolution (frame rate) are high relative to the speedsof the objects depicted in the frame data. However, that ease ofdetection and tracking is completely dependent on analysis on highresolution, high frame rate data including relatively low object speeds.This dependency significantly limits the application and utility ofdetecting and tracking objects in this way.

With the understanding of the operation and shortcomings of conventionalvideo technology (discussed with reference to FIGS. 7-8) in mind,disclosed embodiments are based, at least in part, on NM vision theorywhich recognizes that an image sequence of objects output from aconventonal video camera provides data that may be cummulative andunnecessary for further processing. Developers of conventional analog NMcameras have recognized that the use of NM techniques provide somedegree of improvement upon conventional video cameras because NMprioritizes changing data and de-prioritizes static data.

NM vision is based on the recognition that not all frame data, andmoreover, not all frames include information that is required for allimage processing operations. More specifically, motion vectors may becomputed using only key frames as shown in FIG. 9, which illustratesanalog NM camera operation. As shown in FIG. 9, rays of light fromreflected objects in the environment are focused on an array ofphotoreceptors. Each photoreceptor converts the light energy intoproportional analog intensity values. Analog circuitry may be added toeach pixel to generate an output sequence of spikes on a one-to-onebasis. That circuitry generates a spike whenever the intensity valueexceeds a threshold. Thus, when a spike is generated, the old intensityvalue may be updated with the new intensity value. In accordance withanalog NM technology, each spike can be assigned a spatial-temporalcoordinate (x, y, t, c) where (x, y) is the spatial location of thephotoreceptor in the image, (t) is the timestamp when the threshold wasexceeded, and (c) represents the polarity of the spike.

FIG. 10 provides a diagrammatic view of the operation of a single analogneuromorphic pixel. Similar to a conventional sensor, rays of lightreflected from objects in the environment are focused onto thephotoreceptor. The photoreceptor converts the light energy into ananalog intensity. Additional analog circuitry is added to each pixel tothreshold the analog intensity into analog spikes. The upper graphillustrates how an analog intensity is thresholded. A green spike(depicted with a green arrow) is issued whenever the analog intensitytransitions a threshold with a positive slope and a red spike (depictedwith a red arrow) is issued whenever the analog intensity transitions athreshold with a negative slope.

Analog neuromorphic pixels operate independently of each other andthreshold the log of the analog intensity in order to maximize dynamicrange. An advantage of the analog neuromorphic camera is that higheffective frame rates can be achieved while maintaining a significantlylow data rate because the generated spikes are sparse. The spikes froman analog neuromorphic camera tend to preserve information about therelative movement of the objects in the environment relative to thecamera.

However, conventional, analog NM cameras are only capable of capturingtwo-dimensional (2D) events. This is because, as explained above, analogneuromorphic pixels operate independently of each other and thresholdthe log of the analog intensity in order to maximize dynamic range andthere is a one-to-one correlation between each pixel or photoreceptorused in an analog NM camera and the corresponding software thatsimulates operation of a Retinal Ganglion Cell (RGC). RGCs are neuronslocated near the inner surface of the retina of the eye that receivevisual information from photoreceptors and collectively generateimage-forming visual information from the retina in the form of actionpotential or nerve impulses. In the study of the human eye-brainmachine, that action potential is often referred to simplistically as“spikes;” as a result, the term “spikes is used herein to refer to datagenerated by components simulating operation of RGCs for the purpose ofperforming NM vision.

To the contrary, the presently disclosed embodiments utilize additionalstructure and techniques to enable analysis of NM image data to identifyspatial and temporal data that enables three-dimensional (3D) analysisas well and associated image processing analysis. This is, in part,based on the shift from an analog implementation to a digitalimplementation provided by the disclosed embodiments. That shift enablesand alters the relationship between pixels (photoreceptors) andstructure for synthesizing RGC operation from one-to-one (Pixel-to-RGC)to many to one (Pixels-to-RGC). That shift enables communication andconsideration of data generated by a plurality of pixels when generatingspike data.

That shift alters vision data generation in a manner that is akin to thedifference between how the fovea centralis of the eye operates (fovealvision) and how parts of the eye outside the fovea centralis operate forperipheral vision. More specifically, the human eye includes two typesof photoreceptors: cones and rods. The fovea centralis is a smalldepression in the retina of the eye where visual acuity is highest. Thecenter of the field of vision is focused in this region. The foveacentralis is where retinal cones (photoreceptors for color detection)are particularly concentrated. The fovea centralis does not include rods(photoreceptors more sensitive than cones but unable to detect color).

Conventional, analog NM vision systems operate in the same manner as thefoveal centralis region, wherein there is one-to-one correspondencebetween photoreceptors and RGC. To the contrary, disclosed embodimentsof the digital NM vision system synthesize operation of thephotoreceptors in the region outside the fovea centralis. In that areaof the eye, neurons provided between the photoreceptors and the RGCsenable “cross-talk” or communication and consideration of photoreceptordata from nearby, neighboring and/or near photoreceptors prior to thatdata being used to generate spike data by the RGCs. In the human eye,this “cross-talk” appears to enable generation of different data by theRGCs than is generated by the RGCs in the foveal centralis region.Likewise, the digital NM vision system of presently disclosed generatesdifferent data than that generated by analog NM vision systems in thatthe digital NM spike data is based on more comprehensive data.

FIG. 11 demonstrates the advantages of a digital NM vision systemprovided in accordance with the disclosed embodiments over both aconventional, analog NM camera and a typical video camera. The graphplots the technologies as a function of spatial resolution (frame size,e.g., pixels per square area unit) versus temporal resolution (framerate, e.g., Hertz).

As is known generally, the ease of detection and tracking usingconventional camera technology is completely dependent on analysis onhigh resolution, high frame rate data including relatively low objectspeeds. However, this dependency significantly limits the applicationand utility of detecting and tracking objects because of the largeamounts of data to be analyzed.

For example, most commercially available video systems today are deemedto be high resolution and, therefore, generate a large amount of data.For example, even a gray-scale camera that outputs 2K×2K pixel images,at a rate of 1000 frames per second, requires an output bandwidth of 4GBper second. However, such high frame rates create a problem regardingthe amount of data generated. For example, the gray-scaled camera dataproduces by the equipment illustrated in FIG. 4 would fill a 32 GB SDcard every eight seconds. For color images, such as that produced by theequipment illustrated in FIG. 5, the problem is further exasperatedbecause the required output bandwidth for a color video camera is 12 GBper second.

Moreover, the large quantity of data produced by such conventionalsystems not only causes storage challenges but also challenges regardingprocessor capabilities for analyzing such data in an effective manner.For example, such a large amount of generated data may not be useful forequipment automation, machine learning, driver assistance or autonomousdriving applications if the data cannot be analyzed in a timely mannerto provide direction and/or control.

As explained above, such disadvantages are at least partially addressedby use of analog NM cameras. However, such cameras still require arelatively high effective frame rate. Moreover, analog NM cameras sufferfrom the additional disadvantage of the cost (monetary and spatial) ofadding substantial analog circuitry to each pixel (because it operateson a pixel basis only) in order to generate the sparse events. Thisdisadvantage of requiring additional and custom analog circuitry foreach pixel/photoreceptor limits the spatial coverage and resolution ofthe images as compared to conventional, off-the-shelf image sensors.

With these relative deficiencies and limitations of conventional videoand analog NM imaging in mind, disclosed embodiments provide a digitalimplementation of NM vision image processing that minimizes oreliminates those issues. The shift from analog to digital implementedherein also enables the ability to reduce the data frame rate whileincreasing the acuity provided by the image generated by the visionsystem.

Relatively large frame sizes (e.g., 2048 by 2048 pixels) may be achievedby the digital NM system using “off-the-self” image sensor (e.g., SonyIMX252 Monochrome and Color CMOS Image Sensor) found in typical videocameras. Additionally, relatively high temporal resolution (e.g., 1millisecond) may be achieved by the digital NM system running suchsensors at higher frame rates (e.g., 1000 frames per second) so as toexploit digital processing techniques to extract sparse motion eventsfrom the frames. In this way, the digital NM vision system may include acombination of software running in the digital NM detector 110 (e.g.,the velocity transformer module 140 illustrated in FIG. 1) and one orprocessors running software on back-end to generate digital NM output(e.g., in the digital NM engine 145 of FIG. 1).

In accordance with at least one embodiment, digital NM output mayinclude data generated and analyzed on a pixel-by-pixel basis generatedusing data generated by a plurality of pixels/photoreceptors so as toenable consideration of data generated by a neighborhood ofpixels/photoreceptors. As such, the digital NM output may include spikesthat pertain to an event in space-time that includes a localneighborhood of space-time statistics (e.g., including data indicatingpolarity, edge, images, etc.)

Returning now to the example of the presently disclosed digital NMvision system of FIG. 1, the illustrated example provides a 5D digital,NM vision system in that velocity vectors are computed for each pixel inan image. As shown in FIG. 1, a photoreceptor array 120 converts rays oflight 105 reflected from objects in an environment into a temporalsequence of spatial images 125. A digital retina 130 converts thesequence of images into a sparse sequence of spikes 135. The velocitytransformer module 140 uses the sequence of spikes 135 to generatevelocity vectors 115 for each object in the scene.

FIG. 12 shows a conceptual, illustrative view of an example of digitalNM sensor 1200 (corresponding to the example sensor 120 illustrated inFIG. 1). The illustrative functionality corresponds to at leastcomponents 120 and 130 illustrated in FIG. 1. That digital NMfunctionality may be implemented using a conventional off-the-shelfphotoreceptor array (e.g. CMOS sensor) to output image frames at a highframe rate (e.g., 1000 frames per second) at 1210. The functionality ofthe digital retina 1220 may be augmented with digital circuitry at 1230that compares the incoming image frames to a store image frame. Morespecifically, whenever the intensity values of a pixel of the incomingimage transitions a threshold in the corresponding pixel of the storedimage, a digital spike may be issued and the corresponding receptorfield area of the stored image may be updated to a current value.However, as explained above, that digital circuitry 1230 is furtherconfigured to take into consideration a plurality ofpixels/photoreceptors so as to enable consideration of data generated bya neighborhood of pixels/photoreceptors. As such, the digital NM outputfrom 1230 may include spikes that pertain to an event in space-time thatincludes a local neighborhood of space-time statistics (e.g., includingdata indicating polarity, edge, images, etc.)

It should be understood that the digital retina is not merely a serialviewer of images. Rather, the digital retina may be conceptually thoughtof as a temporal bookkeeper. This is because, every time a referenceimage changes, a spike is generated. Thus, operation of the digitalretina enables documentation of when and how parts of an image change.

FIG. 13 illustrates one example of conceptual components of the digitalretina 130 in further detail. As shown in FIG. 13, the digital retina1300 (corresponding to the example 130 illustrated in FIG. 1) receivesinput 1305 of an input image 1320 from the photoreceptor array whilemaintaining a history of previous input images 1315. The current inputimage 1320 and the previous input images 1325 are processed by an InputTransform 1330 to generate a transformed input image 1335.

Additionally, the current state image 1380 is a reference image thatrepresents the most recent aggregation of all updates extracted from theinput images in the form of spikes. The current state image, at 1385 isinput into and processed by a State Transform 1390 to generate atransformed state image 1340.

The transformed input image 1335 and the transformed state image 1340are compared and thresholded to generate state change data 1350. Eachstate change in the state change data 1350 generates a spike 1355 thatis output in a spike sequence 1360 as well as an update 1365 to thecurrent state image 1380 for subsequent use. More specifically, thatupdate 1365 is processed by an Update Transform 1370 to generate atransformed update 1375. In this way, the transformed update 1375 isused to update the current state image 1380.

FIG. 14 illustrates conceptual, illustrative view of the functionalityprovided by components illustrated in FIG. 13. FIG. 14 provides adiagrammatic view of a polarity spike sequence generated for a verticalwhite bar moving from the left to the right of the image at 1400. Asshown at 1405, each image frame generates six polarity spikes with theleading edge generating three spikes 1406 and the trailing edgegenerating three spikes 1407. Those polarity spikes are used to updatethe stored state image. Each spike updates a single pixel.

The generated spikes, 1406 and 1407, are output as a sequence of eventor digital spikes 1410. Each digital spike is defined by a spatialcoordinate (x, y), timestamp (t), and polarity (p). The spatialcoordinate is the location of the spike on the input image and thetimestamp is derived from the timestamp of the input image frame. Thepolarity specifies whether the intensity of the reference image shouldincrease or decrease. Thus, the digital spikes, 1412 and 1411 are thedigital spikes output for the generated spikes 1407 and 1406 of Frame 0in 1405. The spike sequence 1410 is also used to update the currentstate image. The spikes with polarity updates 1417 and 1416 are used toupdate the current state image at 1422 and 1421, respectively. The greenpolarity spikes 1416 increase the intensity of its corresponding pixelsin the current state at 1421 and the red polarity spikes 1417 decreasethe intensity of its corresponding pixels in the current state image at1422. The updates to the reference image 1422 and 1421 applied to thegenerated spikes of each frame 1407 and 1406 cause the current stateimage to transform to be similar to the input images over time.

In accordance with at least one implementation, the digital NM retinamay be implemented in combination with may be implemented as an imagingsensor and/or sensor array that functions as a digital retina. Thedigital retina in combination with computation devices (for example,CMOS, e.g., FPGA, GPU, etc.) may form a digital NM detector.

FIG. 15 illustrates a transformation of the input image using acenter-on adaptive threshold. The human retina performs center-surroundadaptive thresholding on input images. A center-surround filter 1500 isdefined by an inner ring and an outer ring. The center-surround filteris applied to the input image at 1510. There are two basic varieties offilters: center-on/surround-off, and center-off/surround-on. Forcenter-on, surround off, the intensity of the center pixel is computedby subtracting the average intensity of the pixels in the inner ringfrom the average intensity of the pixels in the outer ring. Forcenter-off/surround-on, the center pixel is computed by subtracting theaverage intensity of the pixels in the outer ring from the averageintensity of the pixels in the inner ring. The resulting output ofapplying a center-surround to an input image is 1505. The resultingmotion patterns of 1505 are similar to that of the original inputimages.

FIG. 16 provides a diagrammatic view of a micro-fovea spike sequencegenerated for the same vertical white bar in FIG. 14 moving from theleft to the right of the image at 1600. As in FIG. 14, output spikes aregenerated at 1605. However, for micro-fovea generation, each image frameessentially replaces the six polarity spikes 1611 with a single polarityspike that is a micro-fovea spike 1612 at 1610. Thus, at 1615, theresulting micro-fovea 1617 surrounds the spike center 1616 (whichcorresponds to the generated micro-fovea spike 1612) and contains all ofthe information about the update to the current state image (e.g. 1350in FIG. 13). In this way, this information 1622 may be attached to thespike 1621 at 1620 and may be used to update the current state image bycomponents of 1300 illustrated in FIG. 13.

As explained briefly above, in accordance with at least one embodiment,velocity vectors may be calculated for each pixel in an image. Inaccordance with disclosed embodiments, such velocity vectors may be usedto generate and analyze spatial-temporal dimensions. More specifically,a digital NM detector may generate a sequence of NM images that areparticularly effective at illustrating spatiotemporal patterns. Forexample, as illustrated in FIG. 17, by rotating and/or skewing velocityspace of the spatio-temporal events reveals clearly visible motionpatterns in the underlying image sequence (e.g., video data). Thus,within a photographic image 1700, velocity vectors 1705 may besuperimposed upon the image 1700 to illustrate the spatio-temporalrelationships between motion depicted in an image sequence. Thosevelocity vectors 1705 may be colored differently depending the speed ofassociated with the vector, for example, the velocity vectors 1710 maybe colored red, indicative of a minimum speed of 0 pixels/50 frames anda maximum speed of 5 pixels/50 frames. As shown in FIG. 20, the velocityvectors 1705 are indicative of the movement associated with thepedestrian's feet.

Thus, in at least one implementation, be differentiating colors of thevectors 1700, it is more readily apparent where the motion patternsoccur. In one implementation, for example, the color red would be asindicated above, with the color yellow corresponding to a minimum of 5pixel/50 frames and a maximum of 10 pixels/50 frames, the color greencorresponding to a minimum of 10 pixel/50 frames and a maximum of 15pixels/50 frames, the color cyan corresponding to a minimum of 15pixel/50 frames and a maximum of 20 pixels/50 frames, the color bluecorresponding to a minimum of 20 pixel/50 frames and a maximum ofinfinity pixels/50 frames with the color yellow corresponding to aminimum of 5 pixel/50 frames and a maximum of 10 pixels/50 frames.

As a result of analyzing the motion patterns, image data may besegmented based on detected velocity vectors. This enables the abilityto better identify objects within a velocity space. With such data, thereference frames may be rotated and/or skewed to more clearly conveyvisible unique motion patterns within the data. Accordingly, suchoperations provide additional utility in that they enable improvedseparation of objects with different motion patterns from within inputimage data. Such technology may be used to provide strong indicators ofocclusion as well as additional functionality detailed herein.

As explained above, the velocity vector data may be used to represent orcharacterize velocity space. That velocity space may, in turn be used toperform velocity segmentation to identify and differentiate objects.Velocity segmentation may be performed in any number of ways includingusing, for example, a feature-based approach that utilizes lower framerate data (e.g., 30 frames per second), a relatively dense method, orsparse method that utilizes faster frame rates (e.g., 1,000 frames persecond) relative to the motion being detected with a velocity beingassigned to each pixel in an image.

FIG. 18 illustrates is an illustrative diagram useful in describing oneexemplary process of constructing a velocity space. As shown in FIG. 18,a temporal sequence of spatial image frames may be analyzed at 1800 toreplace each frame with a corresponding spiking pattern at 1805.Subsequently, the temporal sequence of the spiking patterns 1820 may belayered or stacked in time to form a 3D velocity space (x, y, t) at1815. Such operations enable the ability to view the velocity space fromalternative viewpoints, e.g., the top-view 1820 of the velocity space.Thus, through rotation of the velocity space alternative perspective isprovided. In this way, a velocity of interest may be represented byrotating and/or skewing the velocity space to an angle that representsthat velocity of interest.

In accordance with at least one embodiment, this may be performed byprojecting spikes onto a plane and performing a histogram in spatialbins of the data. This operation may be repeated for all angles ofinterest so as to obtain a complete data set for all velocities ofinterest in the velocity space. As a result, for each spike, a velocityvector is assigned to the bin with the maximum count and maximumtrajectory length across all computed velocity angles.

The utility of such image data generation and processing is illustratedwith reference to FIG. 17. The image data of FIG. 17 was generated at aspeed measured as pixels traveled per 50 frames. The depicted pedestrianis traveling 17 pixels/50 frames to the right. Note green to cyanvelocity vectors passing through spikes from the pedestrian's back. Inthe example of FIG. 17, the velocity space was rotated counter clockwiseor clockwise depending on the direction of motion for the particularpixel. For predominately horizontal motion, the rotation was performedaround the vertical (Y) axis.

For the scene depicted in FIG. 17, the velocity space was rotatedcounter-clockwise at angles corresponding to speed from 5 pixels/50frames to 30 pixels/50 frames in increments of 5 pixels/50 frames. Eachrotation was projected with perspective the spikes into a velocityscreen space.

At each angle, the spikes falling on the same pixel column in thevelocity screen space were counted and input into a histogram. Thevelocity screen space pixels columns with, for example, 80% of themaximum count, may be replaced with a velocity vector from the earliestspike in the column to the latest spike in that column. The resultingvelocity vectors in the velocity space may then be plotted with colorassignments. An example of that is illustrated in FIG. 17. Althoughthere are many ways of associating spike data to determine theirvelocity, FIG. 19 illustrates one example of how that may be performed.FIG. 19 illustrates an example of generating velocity vectors byrotating and/or skewing velocity space. Such velocity vector generationis particularly valuable for characterizing image data because suchvelocity vector data may be used to further identify foveas, anddifferentiate objects within that image data.

Accordingly, as shown in FIG. 19, velocity vectors may be generated inthis way by first dividing a velocity plane 1905 into bins at 1910. Avelocity plane may be conceptually thought of as two-dimensionalhistogram of projected spikes where the spike falling into a histogrambin are assumed generated from a point on an object moving at a speed asspecified by the rotation angles of the velocity space.

As shown in FIG. 19, at the 0 degree angle of rotation, each horizontalblack line represents a frame of video data with a representation of theobject (the circle) from a top view as in FIG. 18. The circles representobject moving from left to right over time (from top to bottom).

A spike sequence may be projected onto the velocity plane 1905 at 1915.Subsequently, the spike sequence may be projected onto the velocityplane 1905 at each of a plurality of rotation angles at 1920. Thus, forexample, at 1925, the velocity space 1915 may be rotated about thespatial and temporal axes. The angle of 45 degrees corresponds to thevelocity of the spikes of the bar moving from left to right in FIG. 18.Since the rotation of the velocity space in 1925 corresponds or is tunedto the velocity of the vertical bar moving from left to right, the countof the spikes accumulating in the velocity plane 1915 will peak. Thusthe peak indicating these spikes are not only emanating from the samepoint on the vertical bar but the spikes are all moving at the samespeed as indicated by rotation angle of the velocity plane.

By rotating the velocity space such that the object representationslines up so as to project on top of each other indicates that the objectThis relationship enables the ability to form velocity vectors. Thus, berotating the angle so that the object lines up

Moreover, as explained above briefly, conventional analog NM camerasoperate on the same principal as the human eye within the foveacentralis, wherein each photoreceptor is associated with and directlyconnected to each RGC to produce a spike that is specific to thatphotoreceptor. This one-to-one relationship creates a limitationhowever, in that analog NM cameras, like the photoreceptors in the foveacentralis of the human eye are unable to differentiate a non-movingobject on a non-moving background. In the human eye, this deficiency isremediated or cured by the photoreceptors and RGC relationship presentin the area of the eye outside the foveal centralis; as explained above,that area includes photoreceptors that are coupled to and communicatingwith RGCs in an indirect manner through a plurality of neurons thatenable the ability to better differentiate a non-moving or slow movingobject from a non-moving or slow moving background.

In the same way, the digital nature of the presently disclosed digitalNM vision system synthesizes the relationship between neurons providedbetween the photoreceptors and the RGCs in the human eye that enable“cross-talk” or communication and consideration of photoreceptor datafrom nearby, neighboring and/or near photoreceptors prior to that databeing used to generate spike data by the RGCs. Thus, the spike datagenerated by the digital NM vision system of presently disclosedgenerates different data than that generated by analog NM vision systemsin that the digital NM spike data is based on more comprehensive data.

In accordance with at least one disclosed embodiment, the spike data maybe augmented or used in combination with image data generated byfiltering incoming image data using a color opposite adaptive threshold.In such an implementation, the use of center surround filters (likecenter-surround receptive fields in the retina of an eye) may be used togenerate image data that may, in turn enable the ability to generatezero-crossings that may be used for edge detection. Such capabilitieshave particular technical utility alone, and when combined with theother functionality described herein because they enable the ability touse the zero-crossing data to identify and utilize root polynomial dataso as to attain sub-pixel accuracy.

As a result, in accordance with at least some embodiments, velocityvectors may be computed by rotating and/or skewing the velocity spaceprovided by even a single frame of image data.

Disclosed embodiments utilize structure and software for velocity vectorand velocity trajectory data generation. In accordance with at leastsome disclosed embodiments, velocity vector trajectories may begenerated based on spatial temporal transformation and projection. Thismay be performed by rotating velocity space to an angle that representsa velocity of interest. Subsequently, spikes may be projected onto aplane and a histogram may be used to graphically represent thedistribution of the data to provide an estimate of the probabilitydistribution using spatial bins. This rotation, projection and sortingmay be performed for all angles of interest. Subsequently, for eachidentified spike, the velocity vector may be assigned to the bin withthe maximum count and maximum trajectory length across all the computedvelocity angles.

In accordance with at least one disclosed embodiment, methodologies areprovided for using spread functions to optimize search for velocityvectors. Thus, optionally, to optimize searching for velocities usingthe above-identified procedure, each spike may be projected to the planeas a two-dimensional spread function with radius (r). A hierarchicalsearch can then be performed using the above-identified procedure with alarge radius to record all bins for all angles with a count exceeding apredetermined threshold. Thereafter, the radius can be reduced and theoperations can be performed until the radius is 1. In at least oneimplementation, a Graphics Processing Unit (GPU) can be used to performthe optimization of searching for velocities by representing each of thetwo-dimensional spread functions by an image in which alpha representsthe function.

In accordance with at least one disclosed embodiment, methodologies areprovided for computing velocity trajectories using affinetransformations. Thus, it should be understood that any of theabove-explained operations performed for generating velocity vectortrajectories from spatial temporal transformation and projection may beperformed using either velocity space rotation or affinetransformations.

A velocity space contains a stack of image frames extracted from a video(output by the camera) in sequential order. Spikes are computed for eachframe. The resulting spikes are projected to a histogram on the velocityplane. The velocity space is rotated to various orientation angles tosearch for angles that result in histogram bins with maximum spikecount. The velocity space angles of maximum spike count directlydetermine the velocity of those spikes.

The above procedure can be further refined in various ways. For example,instead of rotating the entire velocity space to an orientation anglerepresenting a constant velocity, the individual frames within thevelocity space can be transformed based velocity profile of themovement. For example, the image frames can undergo affinetransformations including skewing, rotation, and scale.

In accordance with disclosed embodiments, methodologies are provided forcomputation of velocity vectors by rotating velocity space and usingprevious frames to increase the temporal and spatial resolution ofspikes. As explained above, disclosed embodiments pertain to components,system and methodologies for generating relative velocity vectors usingdigital NM data and components and systems for utilizing those velocityvectors for image processing, object detection, classification, andtracking. In accordance with at least one embodiment, an optionaldifferentiation in data processing may be performed based on adetermination whether an object within an image(s) is moving towards oraway from the digital NM detector as opposed to no movement, movementleft, right, upward, or downward (or along any similar translationalorientation), or rotation relative to the NM detector. For example, forobjects moving towards or away from the digital NM detector previousimage frames and/or digital retina data may be scaled and spreadfunctions may be inversely scaled.

Likewise, in accordance with at least one embodiment, an optionaldifferentiation in data processing may be performed based on adetermination whether an object within an image(s) is moving from leftto right, or from right to left, or in any similar translationalorientation relative to the digital NM detector as opposed no movement,movement towards or away, or rotation relative to the digital NMdetector. For objects moving from left to right and constant velocities,the previous frames and/or the digital retina data may be skewed.Furthermore, optionally, for objects determined to be rotating, previousframes and/or digital retina data may be scaled non-linearly and spreadfunctions may be inversely scaled and rotated to match counterparts in3D space.

It should be appreciated that all other motions that may be detected foridentified objects may involve similar combinations of operations andthe corresponding processing may be performed to effectively identifyvelocity vector trajectory data.

Spikes may be generated by comparing the input images to a retina image.The intensity of each incoming pixel may be compared to itscorresponding retina image pixel. When the incoming pixel intensity isabove or below the retina image intensity within a specified threshold aspike may be generated and the retina pixel intensity may be updatedwith the incoming pixel intensity. Normally, the spiking threshold maybe selected well above the noise floor of the sensor as to generatespikes for large intensity changes thus resulting fewest possible numberof spikes that still capture the motion in the scene. Optionally, thespiking threshold may be adaptively lowered to a boundary of sparsity ofspikes and dense pixels (e.g., just above a noise floor of the image) toproduce ten to one hundred times more spikes.

Also, optionally, in response to identifying a first spike that fires,all spikes within a neighborhood of the radius (r) around the spike canbe inhibited and updated. For example, when optimizing searching forvelocities by projecting spikes to the plane as a two-dimensional spreadfunction with radius (r) or representing each of the two-dimensionalspread functions by an image in which alpha represents the function, thespread function may reconstitute the spikes that were inhibited.

Moreover, in accordance with at least one embodiment, spiking may,optionally be performed similar to above-described adaptive lowering ofthe spiking threshold to the boundary of sparsity of spikes and densepixels except that a spread function may be replaced with an image patchextracted from an incoming image. As an object is moving in a scene,numerous spikes (e.g., 10 to 100× more) may be generated since the spikethreshold has been lowered close to the noise floor. Therefore, as theobject moves across the retina many, spikes would fire. However, if anentire neighborhood of pixels on the retina may be replaced with theircorresponding pixels intensities from the incoming image, the numerousspikes about to fire may be reset and, thus, suppressed. Therefore, thefirst or winning spike may determine which image patch will be updated.Additionally, the image patch can also be attached to (e.g., associatedwith) this winning spike for downstream processing and fovea generation.

In such an implementation the above-described operations for velocityvector trajectory determination can be performed as described exceptthat the histogram or aggregations of the spikes projected into a binfrom a trajectory may be combined by classifying portions of the imagedata as background, boundary, internal, or in-painted. In such animplementation, those portions of the image determined to be compatiblemay be combined. For example, by identifying portions of image data, asexplained above, with similar and/or consistent velocity trajectories,the image data portions may be grouped together to create foveas.

In accordance with disclosed embodiments, methodologies are provided forcompound fovea generation. These spatial temporal collections of imagedata may be composited into a single spatial image for each time step.Thus, given a set of spikes that have been associated with a specificvelocity, there is an implication that the image patches that arecentered about each spike may also be associated to that velocity. Thus,the resulting temporal stack of these image patches match. As a result,the pixels in the spatial image may be labeled as background,silhouette, internal, and in-paintable. In accordance with disclosedembodiments, methodologies velocity segmentation may be performed usingsuch compound foveas.

Such labeling and alignment of foveas using conventional imageprocessing to achieve sub-pixel accuracy, more accurate association, andmore accurate and higher resolution velocity vectors.

As explained above, in accordance with disclosed embodiments,methodologies are provided for generating composite spikes withneighborhood data. For example, in at least one implementation, spikesmay be combined and micro-fovea may be lined together to generate one ormore velocity silhouettes. Alternatively, methodologies may be providedthat generate composite spikes with neighborhood data in a moresimplistic manner.

In accordance with disclosed embodiments, methodologies may be providedfor computing velocity trajectories for fovea pixels. Thus, fovea datamay be used for velocity segmentation and tracking of objects withinimage data generated by the digital NM vision system. More specifically,the pixels in each fovea can be used to segment an object from itsbackground using the pixel labels explained above. Moreover, each pixelin a fovea image can be assigned to a velocity trajectory and associatedwith all previous fovea images in previous frames of image date, e.g.,indicating of past position and movement.

In a further refinement, the aligned image patches can be reprocessedbut a subsequent retina, in which spikes may only be generated based onmotion relative to the associated velocity of temporal image stack. Morespecifically, for example, a first retina may account for global motionof a scene due to camera movement, whereas a secondary retina maygenerate spikes for objects moving in the scene relative to thestationary background known because of the elimination of the cameramovement.

Moreover, 3D reconstruction using velocity segmentation and tracking offoveas (as explained above) may be performed in accordance withdisclosed embodiments. In such methodologies and vector engineimplementation, foveas with pixel labels and velocity trajectories maybe associated with real-world objects from learned experience usingmachine learning algorithms tuned to the fovea images. Accordingly,fovea labels may be subsequently used to retrieve 3D models of objectsin an environment as causing the fovea images. These 3D models may thenbe tuned via pre-specified parameters to match the fovea data to enablethe foveas to be hierarchically linked together to construct hierarchiesof 3D models that constitute 3D objects in the environment includingrigid and non-rigid bodies.

These fovea images may then be mapped to the tuned 3D models to predictfuture spiking patterns based on the allowable trajectories of the 3Dobject in the environment. Additionally, the 3D model(s) can be tweakedto better match the fovea patterns. Accordingly, one potential sideeffect of such mapping may be creation of a more detailed 3D model of a3D object; such an improved 3D model can be added to a modellingparadigm as an update and used for future 3D modelling queries.

Thus, in accordance with at least some embodiments, observed foveapatterns (velocity trajectories, 3D model parameter trajectories, andthe relationship of those parameters relative to all or some subset ofother objects detected in the environment, along with Global PositioningSystem (GPS) and/or map data) may be used to build, and/or update adatabase that can uploaded to cloud and mined and/or referenced.

In accordance with at least some embodiments, for example, thoseillustrated in FIGS. 20-22, the above-described velocity vectortrajectory generation, velocity segmentation and tracking and 3Dreconstruction based on velocity segmentation may be used to performpredictive modelling within a surrounding environment to “see thefuture.”

For example, fovea trajectories generated, as explained above, may beused to predict future locations of foveas. Using such fovea trajectorydata, identified foveas at current locations may be extracted from abackground in image data. Foveas may then be moved to future locationsbased on the predicted location determined by the fovea trajectory data.Holes in the image data created by current fovea extractions that areleft may be filled in by previous background and/or in-painting. Such animplementation has particular technical utility in that obscured partsof an image may be filled in based on the fovea trajectory data. Suchutility enables the ability to analyze 3D image data and track objectsthat may be partially and/or temporarily obscured by other objects thatmay be in a forefront of images in the video sequences.

The utility described above does not require knowledge of the 3D modelsof a vehicle interior environment since only the 2D silhouettes of theactual 3D objects in the scene of the vehicle interior are moved totheir predicted locations. However, the above utility can be extended to3D. For example, a system may be preloaded with a set of 3D models thatmay represent generic objects within a vehicle interior, e.g., seatbelt, air bag, vehicle seat, steering wheel. When a 2D silhouette isextracted from the scene, it may be compared to 2D silhouettes generatedfrom the set of 3D models. The 3D model that matches the 2D silhouetteextracted from the scene of the vehicle interior may then be selected.The orientation and scale of the model may then be fit to the data. Forexample, if the 3D model is a vehicle driver's hand, then the 3D modelmay be rotated to match the 2D velocity of the object in the scene.

The algorithm can be extended for non-rigid objects. In this case, the3D model may be represented by a motion capture where a motion capturedescribes the position of the 3D model for various activities. Forexample a motion capture of a vehicle driving might describe motions ofsteering a steering wheel, interacting with one more controls or userinterface screens present in a vehicle interior, etc. The 2D silhouettesextracted from the 3D model performing these activities may be comparedto the sequence of 2D silhouettes extracted from the scene. Once the 3Dmotion capture is synchronized with the 2D silhouette, futuresilhouettes can be predicted from the motion capture sequence.

Moreover, disclosed embodiments may use intermediate generated digitalNM data and/or generate relative velocity vector data to perform finepattern recognition capabilities measured at close range including headtracking, eye tracking, biometrics, face detection, pupil dilation, eyegaze, color, blink and lid detection, coordinated head and eye tracking,unsupervised presponding, driver texting detection and alert, driverfatigue detection, occupant body gestures including hand gestures, seatbelt gesture detection, driver emotion detection, lip reading,handwriting recognition.

For example, in accordance with disclosed embodiments, eye tracking maybe performed by tracking eye gaze using generation and analysis of threedimensional (3D) foveas. Likewise, 3D foveas may be used to detect andtrack eye-lid movement, determine eye color. In accordance withdisclosed embodiments, eye blink and wink detection may be performed bygenerating and analyzing spatial temporal patterns. Such embodiments mayalso enable components of the system to be used to perform a methodologyfor detecting driver fatigue or driver emotion and generating an alertregarding such a detection. Likewise, in accordance with disclosedembodiments, coordinated head and eye tracking can be effectivelyperformed.

FIG. 20 illustrates one example of how disclosed embodiments may beutilized to perform a method 2000 to track and/or search for movement ofa vehicle driver's head and/or eye(s). As shown in that figure, adigital NM camera provided in accordance with the disclosed embodiments(e.g., see FIG. 1, digital detector 110 optionally in combination withdigital NM engine 145), may generate digital NM data including imagedata showing a driver of a vehicle at 2005. That digital NM data may beused, at 2010, to compute velocity trajectory data for the computedvelocity trajectories of the driver's head. Thereafter, that velocitytrajectory data may be used to locate the driver's eyes in the velocityspace at 2015 and/or performing searching for and/or eye tracking forthe vehicles eyes at 2030 so as to provide this functionality at 2035.Additionally, once the driver's eyes are located in velocity space at2015, operations may be performed to calculate velocity trajectory datafor the driver's eye(s) at 2020 and/or detect blinking of the driver'seye(s) as 2025. Optionally, the eye(s) velocity trajectory data(generated at 2020) and data resulting from the detection of eyeblinking (generated at 2025) may be used as part of the eyetracking/searching operations performed at 2030.

By enabling the ability to track an occupant's, e.g., driver's head andeye movement, various conditions may be identified and monitored for toensure attentiveness and alertness of a driver. For example, detecting adriver lowering their chin, rapid blinking, eyes half way closing, etc.,may be monitored for and detected to trigger output of audio, visual,and/or tactile outputs to the user to check and/or ensure drivingattentiveness and/or alertness, e.g., flickering of cabin lights, audiooutput including queries, vibration of occupant seat or steering wheel.

Further, a driver's confusion or need for information may be identifiedbased on detection of movement of the user's eyes and/or head, e.g.,glancing or looking quickly across dashboard controls in a manner thatindicates the driver is searching for a control or display. Utility maybe provided by, in response to such detected movement, engaging with thedriver using a voice response menu by inquiring with the driver if theyneed assistance identifying particular functionality, e.g., windshieldwiper controls, infotainment controls, a mechanism for initiating aheads up display mode, etc.

Likewise, inattentiveness of an occupant, e.g., a driver, indicated bydetection of the occupant's head turning to view a backseat of thevehicle, may trigger a heightened monitoring state of the vehicle'ssurroundings, wherein audio is output to the occupant indicating apresence of oncoming traffic, lane drift detected by the vehicle, etc.Further, such detection may trigger driver assist and/or autonomousdriving functionality. Accordingly, it should be appreciated thatincreased safety and the occupant's need for increased vehicle safetysupport may be detected based on the digital NM data monitoring of theoccupant.

In accordance with disclosed embodiments, a methodology and componentsof a system may detect and recognize NM signatures of occupants as theyperform physical actions. In implementation, this may be performed usingmonitoring of a vehicle Controller Area Network (CAN bus) in combinationwith detection of digital NM image data performed using a hereindisclosed digital NM camera. This may be performed by the systemrecognizing trajectories of a gesture, where the system is able topredict the outcome of an action before a subject (e.g., driver)completes the action enabling the vehicle to appear to be “presponding,”or pre-responding to an instruction. Optionally, the occupant may not berequired to complete the gesture in order to select the intended actionbased on the vehicle components mapping the intended operation to thebeginning sensed motion for that operation instruction.

For purposes of the present disclosure, a “gesture” may include, forexample, a particular position of a user's hand or a particular movementcarried out with the user's hand. The gestures may be configured in amanner known per se. They comprise, in particular, pointing gestures,swiping gestures and those gestures which are used every day, forexample hand rotations, grasping gestures and combinations of aplurality of such gestures which are possibly carried out in immediateshort succession. The gestures may be carried out in a detection spacewithout the user having to establish a spatial relationship with respectto a display area of a user interface, e.g., touch screen. Inparticular, it is not necessary to approach a particular graphicalobject in the display area. In accordance with disclosed embodiments,controlling a user interface using gestures provides the user with aparticularly simple and intuitive input possibility for controllingdisplay contents in a display area or the associated devices.

Thus, as illustrated in FIG. 21 disclosed embodiments may be utilized toperform one or methodologies 2100 used to identify and/or predict anaction or event associated with movement of an occupant of a vehicledriver. As shown in that figure, a digital NM camera provided inaccordance with the disclosed embodiments (e.g., see FIG. 1, digitaldetector 110 optionally in combination with digital NM engine 145), maygenerate digital NM data including image data showing an occupant, e.g.,a driver of a vehicle at 2110. Simultaneously (or prior to orsubsequent) to that generation of digital NM data, a vehicle CAN bus mayreceive data at 2105 from one or more sensors and/or through interactionof a user with one or more user interface components included in avehicle (e.g., touch screens, switches, knobs). It should be understoodthat, in at least one implementation, the digital NM data may beanalyzed to indicate a user's approach to one of such user interfacecomponents as well or instead. Based on the data received at 2105.

Analysis of the received vehicle CAN bus data at 2115 is performed todetect an action performed by a user with relation to components coupledto the vehicle CAN bus, e.g., the one or more user interface componentsprovided in the vehicle. Detection of such an action at 2115 triggersanalyses of digital NM data generated by the occupant NM camera. Morespecifically, at 2120, digital NM data corresponding to a period of timeprior to the detected action may be analyzed to generate a NM signaturefor the detected action. Thus, at 2125, velocity trajectory data iscomputed for the NM signature and associated with a description of theaction.

Subsequently, the velocity trajectory data is stored and analyzed by oneor more processors included in the vehicle so as to “learn” a predictionsegment 2130. Such processors may be running Artificial Intelligence(AI) algorithms such that the equipment present on the vehicle is ableto determine whether a learned prediction segment is present inmonitored digital NM data showing the occupant, e.g., driver.

Thus, as shown in FIG. 21, the digital NM data continuously generated at2110 may also be analyzed at 2135 to compute velocity trajectory datathat is compared to the velocity trajectory data of the learnedprediction segment to determine whether one or more prediction segmentsare present in the continuously NM data flow produced by the digital NMcamera. Based on that determination, and the determination of whichprediction segments are present, software running on the one or moreprocessors may predict an event based on the monitored NM image data at2145.

Such prediction of an event occurring inside a transportation vehiclemay enable the ability to prespond to the even prior to the occupantcompleting motion to perform the event. For example, if occupant motionindicates that the occupant is reaching for a particular user interfacecomponent, audio instructions or queries may be output via the vehicle'sinfotainment system, e.g., “What destination would you like to set inthe vehicle navigation system?.”

Alternatively, for example, determination that a user has reached fortheir mobile phone to make a telephone call, may trigger audio inquiryand/or instructions from the vehicle's infotainment system, e.g., “wouldyou like to make a telephone call? It would be safer to do this ‘handsfree.’ I can search your contacts by name, please indicate the name ofthe person you would like to call.” This functionality would requirethat the user's mobile phone is in communication with the vehicleinfotainment system (which may be performed in a number ofconventionally understood ways).

Likewise, various different gestures and body part movements may beidentified and monitored for to ensure attentiveness and alertness of adriver. For example, detecting a driver lowering their chin, rapidblinking, eyes half way closing, etc., may be monitored for and detectedto trigger output of audio, visual (e.g., changing lighting in thevehicle interior), and/or tactile (vibration in seating or steeringwheel) outputs to the user to check and/or ensure driving attentivenessand/or alertness, e.g., flickering of cabin lights, audio outputincluding queries, vibration of occupant seat or steering wheel.Further, certain vehicle function operations may be prohibited untildetection of occupant movement has occurred.

In accordance with disclosed embodiments, the components of the systemmay be used to perform a methodology for detecting driver texting,mobile phone use, etc., and generating an alert regarding such behavior.More specifically, generated digital NM data (e.g., NM signatures) maybe used to track the trajectory of a mobile phone and the driver's leftand right hands. In one particular, implementation, the system maydetect when one of the driver's hands is in position over the phone'sscreen and monitor finger movements to detect typing. As a result, adistracted driver alert, e.g., driver texting alert, may be generatedand/or associated safety actions may be performed.

Thus, as illustrated in FIG. 22, disclosed embodiments may be utilizedto perform one or methodologies 2200 used to detect driver interactionwith a mobile phone and generate an alert regarding such and similaroperations. As shown in that figure, a digital NM camera provided inaccordance with the disclosed embodiments (e.g., see FIG. 1, digitaldetector 110 optionally in combination with digital NM engine 145), maygenerate digital NM data including image data showing an occupant, e.g.,a driver of a vehicle at 2205. That digital NM data may be used tocompute velocity trajectories for both the mobile phone at 2210 and thedriver's right and left hands at 2215. Note, as explained above, thecomputation of the velocity trajectories for these objects involvesidentifying the edges of these objects in the digital NM data. Followingdetermination of the velocity trajectories, analysis is performed onthat data to detect occlusion of the hand and phone velocitytrajectories at 2220. For example, when the velocity trajectories of twoobjects (e.g., object A and object B) overlap, one of the two objectswill be in front of and occluding the other object. For example ifvelocity trajectory of object A intersects the velocity trajectory ofobject B and object A is nearer to the camera than object B, then thetrajectory of object A will occlude the velocity profile of object B.Thus the ‘spikes’ associated with the movement of object B willtemporarily be hidden by the ‘spikes’ associated with the movement ofobject A.

Subsequently, based on an indication of occlusion of the hand and phonevelocity trajectories, operations may be performed in 2225, to identifyand learn gestures indicating that the occupant is texting using theirphone or otherwise interacting with the phone in a manner that isunsafe. This learning of gestures is performed in the same or a similarmanner to the learning of gestures discussed with relation to FIG. 21.Based on resulting learned touch screen gestures, the digital NMcamera's continuous flow of data is analyzed driver texting and othersimilar related behavior at 2230. Accordingly, a driver texting alertmay be generated and transmitted to a remote location, e.g., anemployer, a parent, a transportation vehicle rental company, etc. Suchan alert may be sent via a cellular connection. Alternatively, or inaddition, such an alert may be logged in a processor located on thevehicle for subsequent review and/or reporting to an owner of thevehicle, for example, in commercial driving situations.

The various illustrated implementations and functionality discussed inrelation to FIGS. 20-22 are merely illustrations of the technicalutility that can be provided by using digital NM data to enableeffective identification and monitoring of occupant movement within avehicle. The disclosed functionality and operations performed on digitalNM data discussed in FIGS. 20-22 may be performed in one or moreprocessors that may be considered part of the digital NM engine 145illustrated in FIG. 1. Alternatively, the functionality may be in wholeor in part included in other components provided in a vehicle includingone or more computation processing units, processors or the like coupledto memory including software for performing the disclosed functionality.

As should be clear from this disclosure, using digital NM data toprocess image data in a more efficient manner enables the capability todeliver additional infotainment and safety functionality to an occupantin an intelligent manner.

In accordance with disclosed embodiments, the components of the systemmay be used to perform a methodology for lip reading. Such embodimentsmay utilize intermediately-generated NM data, velocity vectors and/or 3Dfovea data. Likewise, in accordance with disclosed embodiments, thecomponents of the system may be used to perform a methodology forhandwriting recognition. Such embodiments may utilizeintermediately-generated NM data, velocity vectors and/or 3D fovea data.

In accordance with at least one embodiment, the disclosed embodimentsmay be used to present objects in rightful position for augmentedreality and Virtual Reality (VR) systems effectively masking visual anddisplay latencies. This may, for example, enable a VR representation ofa person to catch a real object represented in a VR environmentalrepresentation.

In accordance with at least one implementation, the digital NM sensormay be incorporated in a stereo neuromorphic pair of assemblies.Further, in accordance with at least one implementation, the digital NMdetector may be incorporated in a compound camera. In such animplementation, the computational element of each imaging sensor may becoupled to other computational elements of other imaging sensors, e.g.,adjacent sensors. Such computation elements may be configured tocollaborate with other computational elements to provide functionality.

In accordance with at least one implementation, the digital NM detectormay be implemented in a system that utilizes them in parallel with othertypes of sensors. For example, a digital NM detector may be used tocreate a composite image based on the aggregate information from varioussensors In accordance with at least one implementation, the digital NMdetector may be utilized in a dual camera configuration that utilizes ahalf mirror. Additional utility is provided by such an embodiment inthat the dual camera configuration enables powerful combinations andredundancy.

In accordance with at least one embodiment, the hardware andmethodologies may be utilized as an effective method for compressinghigh framerate video, e.g., by analyzing image data to compress the databy capturing differences between a current frame and a one or moreprevious frames and applying a transformation. For example, as explainedabove, in accordance with at least one embodiment, the engine andmethodologies may compress high frame rate video data by performingfeature extraction close to an imaging sensor to generate an encodedversion of image data that includes differences and surrounding spatiotemporal regions for subsequent image processing.

In accordance with at least one embodiment, human eye NM vision may besimulated using a digital implementation that utilizes communication andconsideration of multiple photoreceptor data to generate spike data; asa result, that spike data may be used to compress high frame rate videodata by performing feature extraction close to the digital NM imagingsensor to generate an encoded version of image data that includesdifferences and surrounding spatio-temporal regions for subsequent imageprocessing. Accordingly, the hardware and methodologies may be utilizedas an effective method for compressing high framerate video, e.g., byanalyzing image data to compress the data by capturing differencesbetween a current frame and a one or more previous frames and applying atransformation.

In accordance with at least some disclosed embodiments, the disclosedembodiments may be used to obtain image data and analyze that image datato improve operation, assistance, control and/or analysis of image datain vehicle driving scenarios, for example, but not limited to those usedin driver assist functionality, automated/autonomous drivingfunctionality, and the like.

Indeed, conventional image processing, object detection, classification,and tracking are the most challenging tasks in assisted and autonomousdriving especially in bad environments, bad lighting conditions, and lowfalse positive/negative rates. Disclosed embodiments enable an increasein the speed, robustness and effectiveness in image processing byreducing extraneous data previously necessary to perform objectdetection, classification and tracking. Additional utility is providedas well including image data compression, deep learning capabilitieswith machine learning.

The large quantity of data not only causes storage challenges but alsochallenges regarding processor capabilities for analyzing such data inan effective manner. Such a large amount of generated data is not usefulfor driver assistance or autonomous driving applications if the datacannot be analyzed in a timely manner to provide direction and/orcontrol.

Disclosed embodiments may be implemented in conjunction with componentsof autonomous driving systems and driver assistance systems included inautomotive vehicles. Thus, the utility of the disclosed embodimentswithin those technical contexts is described in detail. However, thescope of the innovative concepts disclosed herein is not limited tothose technical contexts. Therefore, it should be understood that thedisclosed embodiments provide utility in all aspects of image processingand control, analysis and diagnostic systems utilizing image processing.

Although certain embodiments have been described and illustrated inexemplary forms with a certain degree of particularity, it is noted thatthe description and illustrations have been made by way of example only.Numerous changes in the details of construction, combination, andarrangement of parts and operations may be made. Accordingly, suchchanges are intended to be included within the scope of the disclosure,the protected scope of which is defined by the claims.

APPENDIX A for each extracted velocity profile {  for each frame in thevelocity space {   composite_frame = new image ( ) ;   for eachimage_patch {    overlay the image_patch onto the composite_frame.   } }  align the composite image frames based on the current extractedvelocity profile.  for each pixel in the composite image stack {  compute the mean and standard deviation of the pixel intensites.  assign the pixels with low standard deviation to the fore groundobject   declare the remaining pixels to be background.  } }

1. A neuromorphic vision system generating and processing image data,the system comprising: an image sensor comprising a plurality ofphotoreceptors each generating image data for generation of spike data,wherein spike data indicates whether an intensity value measured by thatphotoreceptor exceeds a threshold; a spike data generator that generatesthe spike data based on the image data generated by the plurality ofphotoreceptors, the spike data generator comprising a plurality ofcomputational elements corresponding to the plurality of photoreceptorsof the image sensor, wherein each of the plurality of computationalelements generates spike data for the respective correspondingphotoreceptor based on the image data generated by at least two of theplurality of photoreceptors, wherein the at least two of the pluralityof photoreceptors includes the respective corresponding photoreceptorand a photoreceptor neighboring the respective correspondingphotoreceptor; and a digital neuromorphic engine coupled to the spikedata generator and receiving the generated spike data, the digitalneuromorphic engine including one or more processors running softwareconfigured to generate, based on the spike data, digital neuromorphicoutput data indicative of the image data gathered by the image sensorand to perform object detection, classification and/or tracking based onthe spike data generated by the spike data generator, wherein thedigital neuromorphic engine generates velocity vector and velocitytrajectory data based on the generated digital neuromorphic output dataand analyzes the velocity vector and velocity trajectory data toidentify predictive movement data associated with an occupant action,and wherein the predictive movement data is compared with continuouslymonitored velocity trajectory data to predict the occupant action. 2.(canceled)
 3. (canceled)
 4. (canceled)
 5. (canceled)
 6. (canceled) 7.(canceled)
 8. (canceled)
 9. (canceled)
 10. (canceled)
 11. (canceled) 12.(canceled)
 13. (canceled)
 14. (canceled)
 15. A method for monitoring avehicle interior using neuromorphic vision system to generate andprocess image data, the method comprising: generating image data usingan image sensor comprising a plurality of photoreceptors; generatingspike data using a spike data generator based on the image datagenerated by the plurality of photoreceptors, wherein the spike dataindicates whether an intensity value measured by each photoreceptorexceeds a threshold, wherein the spike data generator comprises aplurality of computational elements corresponding to the plurality ofphotoreceptors of the image sensor, wherein each of the plurality ofcomputational elements generates spike data for the respectivecorresponding photoreceptor based on the image data generated by atleast two of the plurality of photoreceptors, wherein the at least twoof the plurality of photoreceptors includes the respective correspondingphotoreceptor and a photoreceptor neighboring the respectivecorresponding photoreceptor; and generating digital neuromorphic outputdata using a digital neuromorphic engine based on the spike data,wherein the digital neuromorphic output data is indicative of the imagedata gathered by the image sensor; performing object detection,classification and/or tracking based on the spike data generated by thespike data generator; generating velocity vector and velocity trajectorydata based on the generated digital neuromorphic output data; andanalyzing the velocity vector and velocity trajectory data to identifypredictive movement data associated with an occupant action, wherein thepredictive movement data is compared with continuously monitoredvelocity trajectory data to predict the occupant action.
 16. (canceled)17. (canceled)
 18. (canceled)
 19. (canceled)
 20. (canceled) 21.(canceled)
 22. (canceled)
 23. (canceled)
 24. (canceled)
 25. (canceled)26. (canceled)
 27. (canceled)
 28. (canceled)
 29. A transportationvehicle comprising: a vehicle interior including seating for a vehicleoccupant; and a neuromorphic vision system generating and processingimage data of the vehicle interior and the vehicle occupant, the systemincluding: an image sensor comprising a plurality of photoreceptors eachgenerating image data for generation of spike data, wherein spike dataindicates whether an intensity value measured by that photoreceptorexceeds a threshold; a spike data generator that generates the spikedata based on the image data generated by the plurality ofphotoreceptors, the spike data generator comprising a plurality ofcomputational elements corresponding to the plurality of photoreceptorsof the image sensor, wherein each of the plurality of computationalelements generates spike data for the respective correspondingphotoreceptor based on the image data generated by at least two of theplurality of photoreceptors, wherein the at least two of the pluralityof photoreceptors includes the respective corresponding photoreceptorand a photoreceptor neighboring the respective correspondingphotoreceptor; and a digital neuromorphic engine coupled to the spikedata generator and receiving the generated spike data, the digitalneuromorphic engine including one or more processors running softwareconfigured to generate, based on the spike data, digital neuromorphicoutput data indicative of the image data gathered by the image sensorand to perform object detection, classification and/or tracking based onthe spike data generated by the spike data generator, wherein thedigital neuromorphic engine generates velocity vector and velocitytrajectory data based on the generated digital neuromorphic output dataand analyzes the velocity vector and velocity trajectory data toidentify predictive movement data associated with an occupant action,and wherein the predictive movement data is compared with continuouslymonitored velocity trajectory data to predict the occupant action. 30.(canceled)